Calibration apparatus, calibration method, program for calibration, and calibration jig

ABSTRACT

A calibration apparatus which estimates a calibration parameter of an image acquisition apparatus, comprises a calibration jig which includes at least two planes and in which calibration markers having known three-dimensional positions are arranged in each plane based on a predetermined rule. The calibration apparatus further comprises a calibration marker recognition section configured to measure and number in-image positions of the calibration markers in at least one image obtained by photographing the calibration jig by the image acquisition apparatus, and a parameter estimate section configured to estimate the calibration parameters of the image acquisition apparatus by the use of the three-dimensional position and the in-image position of the calibration markers numbered by the calibration marker recognition section.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2002-382225, filed Dec. 27,2002, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a calibration apparatus and acalibration method in which an image acquisition apparatus iscalibrated, a program for calibration for allowing a computer tofunction as the calibration apparatus, and a calibration jig to becombined with the calibration apparatus.

2. Description of the Related Art

A calibration method of an image acquisition apparatus, or cameracalibration is an important technique in image processing. Thistechnique is used especially in three-dimensional measurement or objectrecognition using a camera and object grasping by a robot. By a cameracalibration technique, optical and geometrical characteristics of theimage acquisition apparatus, such as cameras, are specified as describedlater, and the technique is important also as a problem that projectivetransformation parameters of the camera, a position and orientationparameter of the camera in an environment, and the like are estimated.To calculate or estimate these camera calibration parameters, a marker(marker or landmark for the calibration) having a knownthree-dimensional position in a world coordinate frame in which thecamera is disposed is photographed by the camera. A two-dimensionalposition of the image of the marker photographed in the camera image iscalculated in the image. Various camera parameters are calculated from apositional relation between the three-dimensional position of the markerand the two-dimensional position of the image of the marker. Variousmethods of the camera calibration have been known.

One of the methods of the camera calibration is described in J. Weng, P.Cohen, and M. Herniou, “Camera Calibration with Distortion Models andAccuracy Evaluation,” IEEE Transactions of Pattern Analysis and MachineIntelligence, Vol. 14, No. 10, 1992, pp. 965 to 980. In the methoddescribed in this document, a plane board on which a plurality of squarecalibration markers having the same size are arranged, and a z-stage onwhich the plane board is slid in a z-axis direction are prepared ascalibration jigs. Vertices of these squares are recognized by imageprocessing to measure the position or each marker in the image. A zvalue obtained from a z stage is combined with an xy coordinate on theplane board to measure the three-dimensional position in the worldcoordinate frame of each marker. Moreover, for of the correspondingmarker, the three-dimensional position in the world coordinate frame andthe two-dimensional position in a plurality of images are used toestimate the calibration parameters of the image acquisition apparatus.

In B. Girod, et al., edited, Principles of 3D image analysis andsynthesis, Kluwer Academic Publishers, 2000, pp. 29 to 30, a method isdisclosed in which the calibration markers are mutually arranged onthree vertical planes, and the marker is recognized to calibrate theimage acquisition apparatus (camera).

In Oliver Faugeras, Three-Dimensional Computer Vision, MIT Press, 1933,pp. 230 to 235, a calibration jig constituted of two planes (calibrationboards) is proposed. A plurality of square markers are arranged on eachcalibration board, and four vertices of each square are extracted in theimage processing to estimate the position of the marker in the image.Color of these squares constituting the markers is different from thatof background, and therefore a boundary via which two planes can beseparated is clear.

BRIEF SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is provideda calibration apparatus which estimates a calibration parameter of animage acquisition apparatus. The calibration apparatus comprises: acalibration jig which includes at least two planes and in whichcalibration markers having known three-dimensional positions arearranged in each plane based on a predetermined rule; a calibrationmarker recognition section configured to measure and number in-imagepositions of the calibration markers in at least one image obtained byphotographing the calibration jig by the image acquisition apparatus;and a parameter estimate section configured to estimate the calibrationparameters of the image acquisition apparatus by the use of thethree-dimensional position and the in-image position of the calibrationmarkers numbered by the calibration marker recognition section.

According to a second aspect of the present invention, there is provideda calibration apparatus which estimates calibration parameters of aplurality of image acquisition apparatus. The calibration apparatuscomprises: a calibration jig which includes at least two planes and inwhich calibration markers having known three-dimensional positions arearranged in each plane based on a predetermined rule; a calibrationmarker recognition section configured to measure and number in-imagepositions of the calibration markers in at least one image obtained byphotographing the calibration jig by the plurality of image acquisitionapparatus from different positions; and a parameter estimate sectionconfigured to estimate the calibration parameters of the plurality ofimage acquisition apparatus by the use of the three-dimensional positionand the in-image position of the calibration markers numbered by thecalibration marker recognition section.

According to a third aspect of the present invention, there is provideda calibration apparatus which estimates calibration parameters of animage acquisition apparatus including a stereo adapter. The calibrationapparatus comprises: a calibration jig which includes at least twoplanes and in which calibration markers having known three-dimensionalpositions are arranged in each plane based on a predetermined rule; acalibration marker recognition section configured to measure and numberin-image positions of the calibration markers in at least one imageobtained by photographing the calibration jig from different positionsvia the stereo adapter by the image acquisition apparatus including thestereo adapter; and a parameter estimate section configured to estimatethe calibration parameters of the image acquisition apparatus includingthe stereo adapter by the use of the three-dimensional position and thein-image position of the calibration markers numbered by the calibrationmarker recognition section.

According to a fourth aspect of the present invention, there is provideda calibration apparatus which estimates a calibration parameter of animage acquisition apparatus. The calibration apparatus comprises: animage input section configured to input an image obtained byphotographing a calibration jig including at least two planes andincluding calibration markers having known three-dimensional positionsand arranged in each plane based on a predetermined rule by the imageacquisition apparatus; a calibration marker recognition sectionconfigured to measure and number in-image positions of the calibrationmarkers in at least one image photographed by the image acquisitionapparatus and inputted by the image input section; and a parameterestimate section configured to estimate the calibration parameters ofthe image acquisition apparatus by the use of the three-dimensionalposition and the in-image position of the calibration markers numberedby the calibration marker recognition section.

According to a fifth aspect of the present invention, there is provideda calibration method in which a calibration parameter of an imageacquisition apparatus is estimated. The calibration method comprises:inputting an image obtained by photographing a calibration jig includingat least two planes and including calibration markers having knownthree-dimensional positions and arranged in each plane based on apredetermined rule by the image acquisition apparatus; measuring andnumbering in-image positions of the calibration markers in at least oneinputted image photographed by the image acquisition apparatus; andestimating the calibration parameter of the image acquisition apparatusby the use of the three-dimensional position and the in-image positionof the numbered calibration markers.

According to a sixth aspect of the present invention, there is provideda program for calibration which allows a computer to realize thefollowing function in order to estimate a calibration parameter of animage acquisition apparatus. The function comprises: inputting an imageobtained by photographing a calibration jig including at least twoplanes and including calibration markers having known three-dimensionalpositions and arranged in each plane Based on a predetermined rule bythe image acquisition apparatus; measuring and numbering in-imagepositions of the calibration markers in at least one inputted imagephotographed by the image acquisition apparatus; and estimating thecalibration parameter of the image acquisition apparatus by the use ofthe three-dimensional position and the in-image position of the numberedcalibration markers.

According to a seventh aspect of the present invention, there isprovided a calibration jig combined with a calibration apparatus whichestimates a calibration parameter of an image acquisition apparatusbased on an image obtained by photographing the calibration jigincluding a predetermined calibration marker by the image acquisitionapparatus. The calibration jig comprises: at least a plurality ofplanes; and a plurality of types of calibration markers which arearranged in each of the plurality of planes based on a predeterminedrule and whose three-dimensional positions are known.

Advantages of the invention will be set forth in the description whichfollows, and in part will be obvious from the description, or may belearned by practice of the invention. Advantages of the invention may berealized and obtained by means of the instrumentalities and combinationsparticularly pointed out hereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the general description given above and the detaileddescription of the embodiments given below, serve to explain theprinciples of the invention.

FIG. 1 is a diagram showing a basic constitution of a calibrationapparatus according to a first embodiment of the present invention;

FIG. 2 is a diagram showing an example of a calibration jig;

FIG. 3 is a diagram showing another example of the calibration jig;

FIG. 4 is a diagram showing still another example of the calibrationjig;

FIG. 5 is a diagram showing one plane constituting the calibration jig;

FIG. 6 is a diagram showing another example of a calibration marker;

FIG. 7 is a diagram showing an operation flowchart of the wholecalibration apparatus according to the first embodiment;

FIG. 8 is a flowchart of a sub-routine for calculating calibrationparameters;

FIG. 9 is an explanatory view of a positional relation among three largemarkers;

FIG. 10 is a diagram showing a coordinate function with a label of themarker in each plane;

FIG. 11 is a diagram showing another example of one plane constitutingthe calibration jig;

FIG. 12 is a diagram showing a typical example of a photographed image;

FIG. 13 is a flowchart of a marker recognition process;

FIG. 14 is an explanatory view of a method of selecting a proximalellipse;

FIG. 15 is an explanatory view showing update of dx, dy;

FIG. 16 is a diagram showing a display example of a recognition result;

FIG. 17 is a diagram showing a first constitution example of thecalibration apparatus according to a second embodiment of the presentinvention;

FIG. 18 is a diagram showing a second constitution example of thecalibration apparatus according to the second embodiment;

FIG. 19 is a diagram showing that the calibration jig is photographed bythe calibration apparatus according to a third embodiment of the presentinvention;

FIG. 20 is an operation flowchart of the whole calibration apparatusaccording to the third embodiment;

FIG. 21 is a diagram showing that the calibration jig is photographed bythe image acquisition apparatus using a camera including a stereoadapter;

FIG. 22 is a diagram showing a photographed image in this case;

FIG. 23 is a diagram showing an example of the calibration jig accordingto a fourth embodiment of the present invention; and

FIG. 24 is a diagram showing another example of the calibration jigaccording to the fourth embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will hereinafter be described withreference to the drawings.

First Embodiment

As shown in FIG. 1, a calibration apparatus according to a firstembodiment of the present invention includes a calibration jig 10 and anarithmetic unit 12. The calibration jig 10 is photographed by an imageacquisition apparatus 14 to be calibrated, and a calibration pattern isadded to the jig. An image (or a video signal) 16 photographed by theimage acquisition apparatus 14 is inputted into the processing unit 12,and the processing unit processes the inputted image (or the videosignal) 16, and calculates a calibration parameter concerning the imageacquisition apparatus 14. That is, the processing unit 12 functions asan image input section, a calibration marker recognition section, and aparameter estimation section.

It is to be noted that the image acquisition apparatus 14 may be anyapparatus for photographing or inputting the image, such as a videocamera and a digital camera. The arithmetic unit 12 may be ageneral-purpose computer such as a personal computer, or anexclusive-use operation processing apparatus. Furthermore, the image (orthe video signal) 16 may be exchanged between the image acquisitionapparatus 14 and the processing unit 12 via communication. The image mayalso be transferred via electronic media (e.g., a memory, compact flash(registered trademark) memory, floppy (registered trademark) disk, smartmedia (trademark), CD-ROM, magnetoptical disk). The processing unit 12may exist in the image acquisition apparatus 14. In this case, theoperation may also be performed by a hardware circuit or softwareprocessing in the image acquisition apparatus 14.

The calibration jig 10 is constituted in the form of a corner cube inwhich three planes 18 cross at right angles to one another. Each of theplanes 18 may have, for example, a square shape as shown in FIG. 2, atriangular shape as shown in FIG. 3, or a pentangular shape as shown inFIG. 4. Furthermore, although not shown, a circular arc shape may alsobe constituted. These three planes crossing at right angles to oneanother are considered to constitute an xy plane, yz plane, and zx planein a coordinate frame specified by the calibration jig 10. For example,in FIG. 2, a plane 18A positioned in a lower part is defined as an xyplane, a plane 18B positioned right above is defined as a yz plane, anda plane 18C positioned left above is defined as a zx plane. Moreover, atthis time, a straight line via which the xy plane intersects with the yzplane is defined as a y-axis, a straight line via which the yz planeintersects with the zx plane is defined as a z-axis, a straight line viawhich the zx plane intersects with the xy plane is defined as an x-axis,and a point at which three planes intersect with one another is definedas an origin O. By this definition, positional relation among the planesand ordinate system can be determined. That is, the xy plane is definedas a z=0 plane, the yz plane is defined as an x=0 plane, and the zxplane is defined as y=0.

As shown in FIG. 5, circular markers 20 b, 20 s having two types ofsizes exist in one plane 18 constituting the calibration jig 10.Concretely, with the calibration jig 10 constituted of three planes asshown in FIG. 2, three large markers 20 b having different sizes existin the vicinity of the origin at which three planes intersect with oneanother as described above. In each plane 18 of the present calibrationjig 10, white is a background, and markers (large markers 20 b and smallmarkers 20 s) are printed in black. The white background including allthe markers in each plane will hereinafter be referred to as abackground region 22 of markers. Moreover, a background boundary (or abackground boundary curve) 24 exists in black so as to surround thebackground region 22 of markers. The background boundary (or thebackground boundary curve) 24 is important in that a region where themarkers exist is limited, and the use of the background boundary 24 willbe described later in detail.

Here, it is evident that the marker is shown in black, the background isshown in white, and, needless to say, opposite colors may also be used.Two types of markers are distinguished by a size such as the largemarkers 20 b and small markers 20 s, but when a color camera can beused, it is evident that two types of markers can be distinguished bycolor. As shown in FIG. 6, the large marker 20 b may also be constitutedof a white/black double circle. As another constitution, the backgroundregion 22 of markers including the markers is gray, the large markers 20b are white, and the small markers 20 s are black. Instead of using twogradations of density values, a marker constitution may use two or moregradations of density values.

Next, an operation of the calibration apparatus constituted in thismanner will be described with reference to a flowchart of FIG. 7. Thatis, the image acquisition apparatus 14 to be calibrated is set in aposition where the calibration jig 10 can be photographed as describedabove (step S1). Moreover, the calibration jig 10 is photographed withthe image acquisition apparatus 14 (step S2). Here, when the calibrationjig 10 is actually photographed with the image acquisition apparatus 14,the jig is photographed in such a manner that the large markers 20 bexisting in each plane 18 exist within the photographed image, or thelarge markers 20 b are arranged in the middle of the image if possible.This is a photography guiding principle for a photographer. Thepositions of the large markers 20 b in the image, which is thephotography guiding principle, plays an important role in enhancingprecision of marker recognition as described later. This is important,especially when an influence of distortion of the optical lens is large.

The image (or the video signal) 16 of the calibration jig 10photographed in this manner is sent to the processing unit 12.Therefore, the processing unit 12 uses geometric information of thecalibration jig 10, and the image (or the video signal) 16 photographedin the step S2 to calculate the calibration parameter concerning theimage acquisition apparatus 14 (step S3).

A sub-routine for calibration parameter calculation performed in theprocessing unit 12 of the step S3 is shown in FIG. 8. First, a step ofrecognizing the background region of markers is carried out (step S31).That is, three background regions 22 of markers are extracted from theimage (or the video signal) 16 obtained from the image acquisitionapparatus 14. Moreover, the xy, yz, or zx plane corresponding to theextracted region is identified, namely recognized.

Subsequently, a step of marker recognition is carried out (step S32).That is, the regions corresponding to the large and small markers 20 b,20 s arranged in each plane are extracted from the region surroundedwith the background region 22 of markers with respect to the respectiveplanes (xy, yz, zx planes) identified in the step S31. Moreover, theseregions are identified, or labeled as those planes.

It is assumed here that three markers constituting the large markers 20b are denoted with A, B, C as shown in FIG. 9, and the plane 18corresponds to the xy plane (plane 18A) of the calibration jig 10. Inthis case, it can be assumed that A-B specifies an x-axis in the worldcoordinate frame which is a three-dimensional space, and A-C specifies ay-axis in the world coordinate frame which is the three-dimensionalspace. Therefore, when a positional relation among A, B, C in the imageis considered as described later in detail, it is possible to calculatea rough direction vector in the image with respect to the x and y axesof the world coordinate frame. Needless to say, the direction vector isa nearly correct estimated value in the vicinity of a position where A,B, C exist. However, excessively apart from A, B, C, the estimated valueof the direction vector includes an error by the influence of thedistortion of the lens of an optical system (not shown) disposed as animage input section or image acquisition means in the image acquisitionapparatus 14. To solve the problem, the calibration marker is identified(or numbered) in a method described later.

Furthermore, FIG. 10 is a diagram showing the labels and coordinaterelation of the markers in each plane. A group of the markers arenumbered like M₁, M₂, . . . , M_(n) spirally from A=M₀ which is a basepoint in a counterclockwise direction. That is, the large marker B isnumbered as M₄, and the large marker C is numbered as M₈. Moreover, forexample, as shown in FIG. 10, a three-dimensional position coordinate ofeach marker can be granted on the basis of the origin O with respect tothe marker group on the xy plane. For example, assuming that offsets toM₀ from the origin O in x and y directions are both a (meter), and aninterval between the markers in the x and y directions is b (meter), athree-dimensional coordinate value (x, y, z) of each representativemarker can easily be calculated as follows.

Marker M₀ (a, a, 0) Marker M₁ (a + b, a, 0) Marker M₂ (a + b, a + b, 0)Marker M₃ (a, a + b, 0) Marker M₄ (a + 2b, a, 0)It is to be noted that, as shown in FIG. 11, only the large marker 20 bnearest to the origin O may also be constituted of the white/blackdouble circle as shown in FIG. 6, so that the marker A indicating M=0can easily be detected.

By the above-described process of the marker recognition, with respectto each marker M_(i) (i=0, 1, 2, . . . ), the position of a markerthree-dimensional position (x_(i), y_(i), z_(i)) in the world coordinateframe on the basis of the calibration jig 10, and a two-dimensionalposition (u_(i), v_(i)) in the corresponding image can be calculated ormeasured.

Thereafter, the process of parameter calculation is carried out (stepS33). That is, the three-dimensional position (x_(i), y_(i), z_(i)) andtwo-dimensional position (u_(i), v_(i)) of the marker associated witheach other in the step S32 are used to calculate the camera calibrationparameters concerning the image acquisition apparatus 14.

These steps S31, S32, S33 will hereinafter be described in detail.

(Recognition Method of Background Region of Markers)

First, a recognition method of background region of markers of the stepS31 will be described. With respect to the image (or the video signal)16 of the calibration jig 10 photographed with the image acquisitionapparatus 14, the markers 20 b, 20 s are photographed substantially inblack, and the background region 22 of markers is photographedsubstantially in white. Then, an appropriate threshold value is set, andthe image 16 is binarized. Then, the background region 22 of markers isconverted to “1”, elliptic or circular regions indicating the markers 20b, 20 s are converted to “0”, and the region of the background boundary24 is converted to “0”. Here, when a binary region indicating “1” isextracted, three background regions 22 of markers and objects other thanthe calibration jig reflected in white are extracted as the binaryregion (object region) designated by the label “1”. It is to be notedthat three background region 22 of markers are separated via the blackbackground boundary 24, and these background regions 22 of markers cancertainly be distinguished from one another as closed regions in theimage. Typical image photography is shown in FIG. 12. In this case,image regions C, D, E which are the background regions 22 of markers andthe other regions A, B, F are registered as regions indicating label“1”.

In the process of photographing the calibration jig, it does not matterif a part of the jig is not be photographed. Nor does it matter if thebackground is photographed, as shown in FIG. 12.

Next, the background region 22 of markers or only a part of the regionis extracted. The following conditions of the extraction may be used:

-   -   1) at least a part of the background region 22 of markers is        positioned in the vicinity of a center of the image;    -   2) the area is larger than some threshold value; and    -   3) the background regions 22 of markers of the three planes are        adjacent to one another in the vicinity of a middle of the area.        By these extraction conditions, for example, as shown in FIG.        12, the image regions C, D, E are extracted/selected. After        extracting a plurality of background regions 22 of markers in        this manner, the xy, yz, or zx plane corresponding to each of        the regions is determined. For this, the following method is        used.

First, it is assumed that the image regions extracted as the backgroundregions 22 of markers are α, β, γ. Next, assuming that centroidpositions in these image regions are P_(α), P_(β), P_(γ), a meanposition is calculated by the following:

$\overset{\_}{P} = {\frac{1}{3}( {P_{\alpha} + P_{\beta} + P_{\gamma}} )}$Moreover, in consideration of vectors {right arrow over (PP_(α))},{right arrow over (PP_(β))}, {right arrow over (PP_(γ))}, these vectorsare rearranged so as to turn counter-clockwise. Furthermore, among theimages 16, an image whose centroid is positioned in a lowermost portionis assumed as the background region 22 of markers constituting the xyplane, and images constituting the yz and zx planes are selected in suchan order that the vectors turn counterclockwise. Accordingly, threebackground regions 22 of markers can be assumed as the xy, yz, zxplanes. In this case, for example, as shown in FIG. 12, it can berecognized that the image region E corresponds to the xy plane, theimage region D corresponds to the yz plane, and the image region Ccorresponds to the zx plane in the background region 22 of markers.

(Marker Recognition Method)

Next, a method of recognizing the marker in the step S32 will bedescribed. That is, an outline of the method of recognizing the largeand small markers for use in the calibration will be described. Asdescribed above, when the recognition of the background region 22 ofmarkers ends, it becomes important to correctly recognized the large andsmall markers 20 b, 20 s surrounded with the background region. Thelarge and small markers 20 b, 20 s are defined as the circular markers,but are not necessarily photographed as right circles in the image 16photographed by the image acquisition apparatus 14. However, the shapecan be approximated by an ellipse.

FIG. 13 shows a flowchart of this marker recognition process. That is,first, only the elliptic region is extracted from the image 16 (stepS320). This is performed as follows. For each background region 22 ofmarkers, a hole filling binarizing process in the background region 22of markers is performed, and an inclusive region including thebackground region 22 of markers and marker region is prepared. Theinclusive region is divided into a marker region and background regionin the plane by the binarizing process in which an appropriate thresholdvalue is used. Alternatively, region segmentation algorithms such asRahardja/Kosaka are applied. Large and small elliptic regions areextracted in this manner. Here, these elliptic regions are representedby E_(j) (j=0, 1, 2, . . . ).

Next, the large elliptic regions E₀, E₄, E₈ corresponding to M₀, M₄, M₈which are large markers 20 b are selected in accordance with theirgeometric constraints such as their areas (step S321). In this case, itis confirmed whether correspondence can correctly be established inconsideration of geometric constraints among the markers. Thereafter,when the centroid of the large elliptic region E₀ is assigned as P₀,direction vectors dx(P₀) and dy(P₀) in P₀ are calculated from apositional relation between the large elliptic regions E₄, E₈ (stepS322). Initialization of the algorithm ends here.

Thereafter, the ellipse E_(k) corresponding to the marker M_(k) (k=1, 2,. . . , N) is searched in order of the markers M₁, M₂, M₃, . . . as in aspiral order. Therefore, first, after initially setting the value of anindex k to “1” (step S323), the following steps S324 to S329 arerepeated. That is, it is first judged whether or not the index k exceedsN in order to judge whether all the markers to the marker M_(N) arerecognized (step S324). Here, in the presence of the marker in which theindex does not exceed N, that is, which is not recognized yet, themarker M_(j) (j<k) whose correspondence with the ellipse is establishedis collected (step S325). It is to be noted that the vicinity of 5×5indicates that a difference of the coordinate value in the x-axisdirection with respect to the coordinate position (x, y) of the notedmarker M_(k) is within two intervals or that the difference of thecoordinate value in the y-axis direction is within two intervals.Therefore, the total number of markers which exist in the vicinity of5×5 is 24 excluding the noted marker M_(k). The vicinity of 5×5 may bereplaced by any other, such as vicinity of 3×3 or vicinity of 7×7.

Next, the coordinate values of the collected marker group {M_(j)|j<k}and ellipse group {E_(j)|j<k} whose correspondence with the markers isestablished are used to predict the position of the marker M_(k) in theimage. For the prediction, the mean and the bounded region of thepredicted image positions associated with the marker M_(k) are computed.Furthermore, direction vectors dx, dy in the centroid of the ellipsecorresponding to the marker M_(k) are also predicted (step S326).Moreover, the ellipse E_(k) which exists in the bounded region of themarker M_(k) and which is nearest to the mean is selected (step S327).Thereafter, the values of dx(P_(k)), dy(P_(k)) corresponding to thecentroid P_(k) Of the ellipse E_(k) including the correspondence (M_(k),E_(k)) of the marker are updated (step S328). Moreover, after increasingthe value of the index k by “1” (step S329), the process returns to thestep S324.

The steps S324 to S329 are repeated in this manner. When the value ofthe index k exceeds N (step S324), the marker recognition process isended.

The marker recognition algorithm, especially the correspondence(labeling) between the marker and the ellipse will hereinafter bedescribed in detail. That is, the labeling between the marker M_(k) andthe ellipse E_(j) extracted from the image will hereinafter bedescribed. Basically, the ellipses corresponding to the large markers(M₀, M₄, M₈) which are major characteristics among the markers are firstextracted. Moreover, an initial estimated value and origin P₀ of thedirection vectors dx, dy in the image of the x and y-axes directionsspecifying the geometric position of the marker are determined by theellipse group already labelled. Thereafter, the ellipses E₁, E₂, E₃, . .. corresponding to the markers M₁, M₂, M₃, . . . are successivelydetermined or numbered in a recursive manner. This method will bedescribed.

It is to be noted that here terms are defined as follows.

1) Marker M_(k): Large Marker (Large Circular Marker) 5 b and SmallMarker (Small Circular Marker) 5 s

The markers are constituted of the large markers 20 b of M₀, M₄, M₈, andthe other small markers 20 s.

2) Ellipse (Large) and Ellipse (Small) E_(j)

A projected image onto the image associated with marker M_(k) cangenerally be approximated by the ellipse. The ellipse extracted from theimage is represented by the geometric features such as the centroid(centroid coordinates) and the areas.

3) x-axis and y-axis, direction step (sx, sy), and direction vector (dx,dy)

The relative position between the markers is represented by sx, sy.Here, sx, sy denote the step numbers in the x-axis and y-axisdirections, and dx, dy denote distances between landmarks disposedadjacent to each other in the image in the x and y directions. Moreconcretely, a distance in which the corresponding ellipse advances byone step in the x-axis direction is dx, and a distance in which theellipse advances by one step in the y-axis direction is dy. Since dx, dyare not uniform in the image, they are defined with respect to thecentroid of each extracted ellipse.

By this definition, the numbering algorithm steps will hereinafter bedescribed.

1) Recognition of Three Large Ellipses (Process of the Step S321)

First, a set of ellipses region-extracted by the process of the stepS320 is sorted by the area in the decreasing order.

Next, three ellipses having a large area are selected, and it is judgedwhether or not these ellipses satisfy the following conditions.

-   -   i) The area is approximately equal (e.g., a region in which        maximum and minimum ratios are present among the areas of three        ellipses).

When this condition is satisfied, three ellipses are assumed to berecognized. Moreover, these three ellipses are numbered as M₀, M₄, M₈.

2) Initial Value Estimate of Direction Vectors dx, dy

An initial value of the direction vector dx which specifies the xdirection between adjacent lattice points is calculated from thecoordinates of M₀, M₄, (u₀, v₀) and (u₄, v₄)

${dx} = {\frac{1}{2}( {\begin{bmatrix}u_{4} \\v_{4}\end{bmatrix} - \begin{bmatrix}u_{0} \\v_{0}\end{bmatrix}} )}$

Moreover, the initial value of the direction vector dy which specifiesthe y direction between the adjacent lattice points is calculated fromthe coordinates of M₀, M₄, (u₀, v₀) and (u₈, v₈)

${dy} = {\frac{1}{2}( {\begin{bmatrix}u_{8} \\v_{8}\end{bmatrix} - \begin{bmatrix}u_{0} \\v_{0}\end{bmatrix}} )}$

3) Recognition of Center Point P₀ (Process of the Step S322)

A large ellipse nearest to the centroid of three large ellipses islabeled as E₀, and the centroid of E₀, (u₀, v₀), is set to the point P₀.The above-described initial values of dx, dy are used as the directionvectors dx(P₀), dy(P₀) of this point P₀.

4) Recursive Ellipse Labeling Method

It is now assumed that an ellipse E_(k-1) corresponding to a markerM_(k-1) and center point P_(k-1) (k=1, 2, . . . ) are obtained. It isalso assumed that direction vectors dx(P_(k-1)), dy(P_(k-1)) in thecenter point of each ellipse can be calculated. In this case, there isalso a possibility that the ellipse E_(i) (0<i<k) corresponding to thesome marker M_(i) cannot be detected. At this time, it is consideredthat the ellipse E_(k) corresponding to the marker M_(k) is labeled.

a) The landmark in the vicinity of 5×5 of the marker M_(k) is collected(process of the step S325).

b) The center point. P_(j) of the ellipse E_(j) already labeled inaccordance with the vicinity marker M_(j) and the direction vectorsdx(P_(j)), dy(P_(j)) are used to calculate a predicted position P_(k) ⁰of the center value P_(k) corresponding to the marker M_(k) (process ofthe step S326). In this case, assuming that steps between M_(i) andM_(k) in the x and y directions are sx, sy (sx=−2, −1, 0, 1, 2; sy=−2,−1, 0, 1, 2) associated with marker M_(i), in the vicinity of MarkerM_(k), the following results, where N_(b)(M_(k)) for M_(k) indicates theset of neighborhood markers for M_(k):

$P_{k}^{0} = {\frac{1}{{N_{b}( M_{k} )}}{\sum\limits_{i \in {N_{b}{(M_{k})}}}\begin{bmatrix}( {{{{sx}( {M_{i}->M_{k}} )}{{dx}( P_{i} )}} + P_{i}} ) \\( {{{{sy}( {M_{i}->M_{k}} )}{{dy}( P_{i} )}} + P_{i}} )\end{bmatrix}}}$${{dx}^{0}( P_{k}^{0} )} = {\frac{1}{{N_{b}( M_{k} )}}{\sum\limits_{i \in {N_{b}{(M_{k})}}}^{\;}{{dx}( P_{i} )}}}$${{dy}^{0}( P_{k}^{0} )} = {\frac{1}{{N_{b}( M_{k} )}}{\sum\limits_{i \in {N_{b}{(M_{k})}}}{{dy}( P_{i} )}}}$

c) Selection of Nearest Ellipse E_(k) (process of the step S327)

The ellipse nearest to the predicted position P_(k) ⁰ of the markerM_(k) is obtained. In this case, only an ellipse in which a distancebetween the predicted position P_(k) ⁰ and the ellipse satisfies thefollowing is taken into consideration:dist<β√{square root over ({dx ⁰(P _(k) ⁰)}² +{dy ⁰(P _(k) ⁰)}²)}{squareroot over ({dx ⁰(P _(k) ⁰)}² +{dy ⁰(P _(k) ⁰)}²)}For example, β=1/2√{square root over (2)} (this is shown in FIG. 14).

d) Update of dx(P_(k)), dy(P_(k)) (process of the step S328)

For a set of the marker and ellipse in which ellipse correspondence issuccessful in the vicinity of 5×5 of the marker M_(k) corresponding toE_(k), the set whose step is only in the x or y direction. That is, onlythe vicinity marker M_(j) in which only one of sx(P_(j)), sy(P_(j)) iszero and whose corresponding ellipse E_(j) is established is selected,and dx, dy are updated by the mean value. (This is shown in FIG. 15)

In accordance with the above-described method, the elliptic markercorresponding to the large and small markers M₀, M₁, . . . in thebackground region 22 of markers can be extracted from the image.Assuming that the three-dimensional coordinate of the marker M_(i) is(x_(i), y_(i), z_(i)), and the image position corresponding to themarker M_(i) is (u_(i), v_(i)), information R of the marker obtainedfrom the calibration jig 10 can be obtained as a pair of thethree-dimensional coordinate value (x_(i), y_(i), z_(i)) in the worldcoordinate frame and (u_(i), v_(i)) obtained from the two-dimensionalimage. Here, it is further important that all the markers M_(i) in thebackground region 22 of markers are not necessarily detected from theimage. That is, as represented by the following, only the detectedmarker group is represented as the set.R={(x_(i), y_(i), z_(i); u_(i), v_(i))|i=1, 2, 3, . . . , n}

FIG. 16 is a diagram showing a recognition result displayed in a displayscreen 26 of a display device disposed in or connected to the arithmeticunit 12. When the results are spirally numbered, as shown in the figure,the number connects the centroids of the markers disposed geometricallyadjacent to each other via a dotted line (or a straight line). Then, anoperator or a user can visually judge whether the results are correctlynumbered. Needless to say, as shown in the figure, the markers disposedgeometrically adjacent to each other are not connected to each other viaa dotted line or a straight line. Additionally, even when the number ofeach marker is added, the effect is apparently obtained.

Moreover, in the above description, the vicinity of 5×5 has been used asthe vicinity of the marker, but the vicinity of 3×3 or 7×7 may also beused.

Furthermore, a method of using the in-image position P_(j)(u_(j), v_(j))of the marker M_(j) in the vicinity which has heretofore been identifiedwill mainly be described in order to predict the in-image positionP_(k)(u_(k), v_(k)) of the marker M_(k). However, this prediction ispossible, even if the in-image position P_(j)(u_(j), v_(j)) is notdirectly used. For example, there is a method of using a radial distancer and angle θ of each marker. That is, the following relational equationmay be used to predict the radial distance and angle (r_(k), θ_(k)) ineach marker M_(k):

$\{ \begin{matrix}{r = \sqrt{u^{2} + v^{2}}} \\{\theta = {\tan^{- 1}\frac{v}{u}}}\end{matrix}\quad $Since this method is similar to a method of predicting the positionP_(k)(u_(k), v_(k)) in the image, the method is not described here indetail.

(Parameter Calculation Method)

Next, a method of calculating the parameter in the step S33 will bedescribed. As described in the previous paragraph, the markerinformation R recognized by the marker recognition method can berepresented as a set of pairs of a three-dimensional point (x_(i),y_(i), z_(i)) and the corresponding two-dimensional point (u_(i),v_(i)). In this paragraph, a method of using these points to calculatethe calibration parameter of the image acquisition apparatus 14 will bedescribed.

First, the definition of the camera calibration will be described. Withrespect to the three-dimensional point (x, y, z) of the world coordinateframe W, coordinate conversion into a camera coordinate frame C isrepresented using Rotation Matrix R=(r_(ij)) and translation vectorT=[t_(x), t_(y), t_(z)]^(t). At this time, when the point on the cameraimage plane is represented by (u′, v′) including the distortion, therelation can be represented by the following:

$\begin{matrix}{{u_{p} = {\frac{u - u_{0}}{\alpha_{u}} = \frac{{r_{11}x} + {r_{12}y} + {r_{13}z} + t_{x}}{{r_{31}x} + {r_{32}y} + {r_{33}z} + t_{z}}}}{v_{p} = {\frac{v - v_{0}}{\alpha_{v}} = \frac{{r_{21}x} + {r_{22}y} + {r_{23}z} + t_{y}}{{r_{31}x} + {r_{32}y} + {r_{33}z} + t_{z}}}}\begin{matrix}{u_{d} = {u_{p} + {( {g_{1} + g_{3}} )u_{p}^{2}} + {g_{4}u_{p}v_{p}} +}} \\{{g_{1}v_{p}^{2}} + {k_{1}{u_{p}( {u_{p}^{2} + v_{p}^{2}} )}}} \\{v_{d} = {v_{p} + {g_{2}u_{p}^{2}} + {g_{3}u_{p}v_{p}} +}} \\{{( {g_{2} + g_{4}} )v_{p}^{2}} + {k_{1}{v_{p}( {u_{p}^{2} + v_{p}^{2}} )}}}\end{matrix}{u^{\prime} = {{\alpha_{u}u_{d}} + u_{0}}}{v^{\prime} = {{\alpha_{v}v_{d}} + v_{0}}}} & ( {{above}\mspace{14mu}{equation}\mspace{14mu}({E1})} )\end{matrix}$At this time, the parameter to be estimated in the camera calibrationunder this representation can be represented as a 15-dimensionalparameter as follows:

$p = \begin{bmatrix}{\alpha_{u},\alpha_{v},u_{0},{v_{0};\phi_{x}},\phi_{y},\phi_{z},t_{x},t_{y},{t_{z};}} \\{k_{1},g_{1},g_{2},g_{3},g_{4}}\end{bmatrix}^{t}$The following is intrinsic parameters of the camera:p_(int)=[α_(u), α_(v), u₀, v₀; k₁, g₁, g₂, g₃, g₄]^(t)This is referred to as extrinsic parameters as a parameter whichspecifies the position of the camera with respect to the worldcoordinate frame:p_(ext)=[φ_(x), φ_(y), φ_(z), t_(x), t_(y), t_(z)]^(t)In the camera calibration, the three-dimensional point group (x_(i),y_(i), z_(i)) and the image corresponding point (u_(i)′, v_(i)′) (i=1,2, . . . , n) are used to estimate a calibration parameter p.

Moreover, in these defining equations, (α_(u), α_(v), u₀, v₀) areparameters associated with the pinhole camera model, and (k₁, g₁, g₂,g₃, g₄) are referred to as parameters associated with the lensdistortion. Needless to say, the distortion parameter of the lens mayalso include parameters (k₂, k₃, . . . ) dealing with higher-orderdistortion. However, in the method described below, it is apparent thateven a high-order or low-order distortion is not related to the essenceof the present invention.

Furthermore, these calibration parameters p are used as terms equivalentto optical parameters of the image acquisition apparatus in the presentinvention.

In this definition, a concrete method is as follows.

1) First, some or all of the three-dimensional point group (x_(i),y_(i), z_(i)) and the image corresponding points (u_(i)′, v_(i)′) (i=1,2, . . . , n) are used to estimate pinhole parameters P₀ in theparameters p:p₀=[α_(u), α_(v), u₀, v₀; φ_(x), φ_(y), φ_(z), t_(x), t_(y), t_(z)]^(t)

2) Next, the above-described pinhole parameter P₀ is combined with aninitial estimated value d₀=[0, 0, 0, 0, 0]^(t) of the parameterconcerning the lens distortion d=[k₁, g₁, g₂, g₃, g₄]t to prepare theinitial value P₀ of p:

$\begin{matrix}{p_{0} = \lbrack {p_{p};d_{0}} \rbrack^{t}} \\{= \lbrack {\alpha_{u},\alpha_{v},u_{0},{v_{0};\phi_{x}},\phi_{y},\phi_{z},t_{x},t_{y},{t_{z};0},0,0,0,0} \rbrack^{t}}\end{matrix}$

3) The three-dimensional point group (x_(i), y_(i), z_(i)), the imagecorresponding point (u_(i)′, v_(i)′) (i=1, 2, . . . , n), and theinitial value p₀ of p are used to update the estimated value p by aniterative method. In the iterative method, the Kalman Filter may beused. That is, (u_(p), v_(p)) is used as the intermediate parameter, thefollowings are used as constraint equations, and the estimated value maybe updated:

$u_{p} = \frac{{r_{11}x_{i}} + {r_{12}y_{i}} + {r_{13}z_{i}} + t_{x}}{{r_{31}x_{i}} + {r_{32}y_{i}} + {r_{33}z_{i}} + t_{z}}$$v_{p} = \frac{{r_{21}x_{i}} + {r_{22}y_{i}} + {r_{23}z_{i}} + t_{y}}{{r_{31}x_{i}} + {r_{32}y_{i}} + {r_{33}z_{i}} + t_{z}}$${f \equiv \begin{bmatrix}f_{1} \\f_{2}\end{bmatrix}} = {\begin{bmatrix}{\alpha_{u}( {u_{p} + {( {g_{1} + g_{3}} )u_{p}^{2}} + {g_{4}u_{p}v_{p}} + {g_{1}v_{p}^{2}} +} } \\{ {k_{1}{u_{p}( {u_{p}^{2} + v_{p}^{2}} )}} ) + u_{0} - u^{\prime}} \\{\alpha_{v}( {v_{p} + {g_{2}u_{p}^{2}} + {g_{3}u_{p}v_{p}} + {( {g_{2} + g_{4}} )v_{p}^{2}} +} } \\{ {k_{1}{v_{p}( {u_{p}^{2} + v_{p}^{2}} )}} ) + v_{0} - v^{\prime}}\end{bmatrix} = 0}$Concretely, the initial estimated value of p is used as the predictedposition to represent mean and covariance matrix ( p, S) as the KalmanFilter constraint equation. Moreover, a measurement value and errorcovariance matrix with respect to each measurement value q=(u_(i)′,v_(i)′) are represented by ( q, Q). Then, with respect to eachthree-dimensional point (x_(i), y_(i), z_(i)) and the imagecorresponding point (u_(i)′, v_(i)′), mean and covariance matrix p:( p,S), which is the second-order statistics of p, is updated to thefollowing P_(new): ( P _(new), S_(new)) by the Kalman Filter. (In thiscase, f is linearized around the estimated value, and accordingly aso-called Extended Kalman Filter is used.) That is, the followingequation group is used:

$\begin{matrix}\begin{matrix}(1) & {M = \frac{\partial f}{\partial p}} \\(2) & {G = {\frac{\partial f}{\partial q}{Q\lbrack \frac{\partial f}{\partial q} \rbrack}^{t}}} \\(3) & {K = {S\mspace{14mu}{M^{t}( {G + {M\mspace{14mu} S\mspace{14mu} M^{t}}} )}^{- 1}}} \\(4) & {{\overset{\_}{p}}_{new} = {\overset{\_}{p} - {K\mspace{11mu} f}}} \\(5) & {S_{new} = {( {I - {K\mspace{14mu} M}} )\mspace{14mu} S}} \\(6) & {{{\text{Process for the next update}\overset{\_}{P}} = {\overset{\_}{P}}_{new}},} \\\; & {S = S_{new}}\end{matrix} & ( {{above}\mspace{14mu}{equation}\mspace{14mu}({E2})} )\end{matrix}$While i=1, 2, . . . , n is changed with (x_(i), y_(i), z_(i)) and theimage corresponding point (u_(i)′, v_(i)′), the estimated value p issuccessively updated, and a mean value p of p and estimated errorcovariance matrix S can be updated.

This method is described, for example, in Y. Motai and A. Kosaka, “SmartView: Hand-Eye Robotic Calibration for Active Viewpoint Generation andObject Grasping,” Proceedings of IEEE International Conference onRobotics and Automation, Seoul, Korea, pp. 2183 to 2190, May 2001, or A.Kosaka and A. C. Kak, “Fast Vision-Guided Mobile Robot Navigation UsingModel-Based Reasoning and Prediction of Uncertainties,” Computer Vision,Graphics, and Image Processing—Image Understanding, Vol. 56, No. 3, pp.271 to 329, 1992, and is not described in detail here.

Moreover, the equation (E1) has heretofore been described, but there isanother method in modeling the camera. For example, in the methoddisclosed in the document of J. Weng., et al., an intermediate parameter(u_(d), v_(d)) is used between the three-dimensional point (x_(i),y_(i), z_(i)) and the corresponding image point (u_(i)′, v_(i)′) toestablish the following constraint equation:

$\begin{matrix}{\;{u_{d} = \frac{u_{i}^{\prime} - u_{0}}{\alpha_{u}}}} & ( {{above}\mspace{14mu}{equation}\mspace{14mu}({E3})} ) \\{{v_{d} = \frac{v_{i}^{\prime} - v_{0}}{\alpha_{v}}}\begin{matrix}{f_{1} = {\frac{{r_{11}x_{i}} + {r_{12}y_{i}} + {r_{13}z_{i}} + t_{x}}{{r_{31}x_{i}} + {r_{32}y_{i}} + {r_{33}z_{i}} + t_{z}} -}} \\{( {u_{d} + {( {g_{1} + g_{3}} )u_{d}^{2}} + {g_{4}u_{d}v_{d}} +} } \\ {{g_{1}v_{d}^{2}} + {k_{1}{u_{d}( {u_{d}^{2} + v_{d}^{2}} )}}} ) \\{= 0}\end{matrix}\begin{matrix}{f_{2} = {\frac{{r_{21}x_{i}} + {r_{22}y_{i}} + {r_{23}z_{i}} + t_{y}}{{r_{31}x_{i}} + {r_{32}y_{i}} + {r_{33}z_{i}} + t_{z}} -}} \\{( {v_{d} + {g_{2}u_{d}^{2}} + {g_{3}u_{d}v_{d}} +} } \\ {{( {g_{2} + g_{4}} )v_{d}^{2}} + {k_{1}{v_{d}( {u_{d}^{2} + v_{d}^{2}} )}}} ) \\{= 0}\end{matrix}} & \;\end{matrix}$Therefore, the Kalman Filter represented by equation (E2) can be used tosimilarly estimate p.

In the above method, error component included in the measuredtwo-dimensional point (u_(i)′, v_(i)′) are regarded as relatively small,and estimated. While this Kalman Filter is applied to each measurementpoint, it is also possible to explude an outlier of measurements witherror. This is executed by checking whether or not the distance betweenthe predicted position and the measured position associated with theprojections of Marker (x_(i), y_(i), z_(i)) onto the image plane exceedssome threshold value, given a current estimate of the parameter p. Thismethod is also disclosed in detail in the document of Kosaka, et al.,and is therefore not described here in detail.

As described above, when the boundary (background boundary 24) of theplane 18 constituting the calibration jig 10 is added, the recognitionof each plane 18 is facilitated. Since special markers (different sizesor colors) are disposed in some of the calibration markers present inthe plane, the numbering (identification) of the markers is facilitated.Therefore, a probability that the outlier by wrong recognition isincluded in the data necessary for the calibration is very small, andthe calibration parameter can be estimated or calculated in a robustmanner. While the calibration parameter is estimated, for example, ithas also an important effect seen in the present embodiment that thecalibration parameter can be estimated with a small numeric calculationamount by the use of the Kalman Filter.

Second Embodiment

In the first embodiment, the calibration jig 10 having a corner cubeshape has been photographed only once to estimate the calibrationparameter p. In the method described in the present embodiment, toperform more precise estimate, the image acquisition apparatus 14 or thecalibration jig 10 is relatively moved and photographed multiple timesduring the photography of the calibration jig 10, and the calibrationparameter is estimated. In this case, as a method of moving the imageacquisition apparatus 14, an operator may manually move the imageacquisition apparatus 14, or special moving sections (e.g., X stage,rotary stage, or robot) may also be used. That is, in the followingmethod, a detailed movement parameter concerning this movement may notbe known. Therefore, the method having further degree of freedom can beproposed, and an apparatus constitution is simplified as compared withthe method described in the document of Motai, et al.

FIG. 17 is a diagram showing a first constitution example of the movingsection. In this constitution example, an image acquisition apparatusmoving device 28 for changing viewpoints of the image acquisitionapparatus 14 by the movement of the image acquisition apparatus 14 isadded to the constitution of the first embodiment. Moreover, for theimage acquisition apparatus 14, processing unit 12, and imageacquisition apparatus moving device 28, timings of movement,photography, and calculation are controlled by a control device 30.These timings are determined by control signals 32, 34, 36.

On the other hand, FIG. 18 is a diagram showing a second constitutionexample in which a calibration jig moving unit 38 for moving thecalibration jig 10 is added. In this case, for the image acquisitionapparatus 14, processing unit 12, and calibration jig moving unit 38,the timings of the movement, photography, and calculation are controlledby the control device 30. These timings are determined by controlsignals 40, 42, 44.

As in these first and second constitution examples, to photograph thecalibration jig 10 from different viewpoints is especially effective,when the number of markers (or three-dimensional points) for thecalibration included in the calibration jig is small. A method ofestimating the calibration parameter by a plurality of measurements willhereinafter be described. Here, to photograph the jig a plurality oftimes, the photography from free viewpoints may be performed, as long asat least a large marker 20 b is photographed with respect to a cornercube.

(1) First, the calibration jig 10 is photographed from each viewpoint k,the three-dimensional point (x_(i) ^(k), y_(i) ^(k), z_(i) ^(k)) andcorresponding image point (u_(i)′^(k), v_(i)′^(k)) are measured, and acalibration parameter p^(k) concerning the viewpoint is estimated. Thismethod is similar to that of the first embodiment. The parameter to beestimated in this manner is as follows:

$p^{k} = \begin{bmatrix}{\alpha_{u}^{k},\alpha_{v}^{k},u_{0}^{k},{v_{0}^{k};\phi_{x}^{k}},\phi_{y}^{k},\phi_{z}^{k},t_{x}^{k},t_{y}^{k},{t_{z}^{k};}} \\{k_{1}^{k},g_{1}^{k},g_{2}^{k},g_{3}^{k},g_{4}^{k}}\end{bmatrix}^{t}$

(2) Next, the calibration parameters from a plurality of viewpoints areput together to produce the parameter p to be estimated:

$p = \begin{bmatrix}{\alpha_{u},\alpha_{v},u_{0},{v_{0};k_{1}},g_{1},g_{2},g_{3},{g_{4};}} \\{\phi_{x}^{1},\phi_{y}^{1},\phi_{z}^{1},t_{x}^{1},t_{y}^{1},{t_{z}^{1};\ldots}\mspace{11mu},\phi_{x}^{m},\phi_{y}^{m},\phi_{z}^{m},t_{x}^{m},t_{y}^{m},t_{z}^{m}}\end{bmatrix}^{t}$In the above, there are camera intrinsic parameters (intrinsicparameters) P_(int) which are common to p^(k) and extrinsic parametersP_(ext) which are not common to p^(k) as follows:p_(int)=[α_(u), α_(v), u₀, v₀; k₁, g₁, g₂, g₃, g₄]^(t)

$p_{ext} = \begin{bmatrix}{\phi_{x}^{1},\phi_{y}^{1},\phi_{z}^{1},t_{x}^{1},t_{y}^{1},{t_{z}^{1};}} \\{\ldots\mspace{11mu},\phi_{x}^{m},\phi_{y}^{m},\phi_{z}^{m},t_{x}^{m},t_{y}^{m},t_{z}^{m}}\end{bmatrix}^{t}$Here, as the initial estimated value of p, the intrinsic parameters forarbitrary viewpoints are used concerning the following, and a valueobtained from each viewpoint is used concerning extrinsic parameters.p_(int)=[α_(u), α_(v), u₀, v₀; k₁, g₁, g₂, g₃, g₄]^(t)

(3) Moreover, the three-dimensional point (x_(i) ^(k), y_(i) ^(k), z_(i)^(k)) photographed in each viewpoint k and the corresponding image point(u_(i)′^(k), v_(i)′^(k)) are used again to update the following:

$p = \begin{bmatrix}{\alpha_{u},\alpha_{v},u_{0},{v_{0};k_{1}},g_{1},g_{2},g_{3},{g_{4};}} \\{\phi_{x}^{1},\phi_{y}^{1},\phi_{z}^{1},t_{x}^{1},t_{y}^{1},{t_{z}^{1};\ldots}\mspace{11mu},\phi_{x}^{m},\phi_{y}^{m},\phi_{z}^{m},t_{x}^{m},t_{y}^{m},t_{z}^{m}}\end{bmatrix}^{t}$For example, when the number of viewpoints is m, the number ofdimensions of p is 9+6m. The constraints with 2 degrees of freedom areestablished with respect to the image point corresponding to eachthree-dimensional point in the same manner as in equation (E3). Theseconstraints are used as the constraint equations of the Kalman Filter.Then, the parameter p can be updated in the same manner as in the firstembodiment.

When the parameters obtained from multiple viewpoints are used by theuse of the above-described steps, the camera calibration parameter canmore precisely be estimated.

Third Embodiment

In a third embodiment, one calibration jig 10 is used to simultaneouslycalibrate a plurality of cameras. FIG. 19 is a diagram showing that thecalibration jig 10 is photographed by the image acquisition apparatus towhich the calibration apparatus according to the third embodiment isapplied. The image acquisition apparatus 14 is constituted of twocameras 46, 48.

FIG. 20 is a diagram showing an operation flowchart of the wholecalibration apparatus according to the present embodiment in which twocameras 46, 48 are simultaneously calibrated. That is, the imageacquisition apparatuses (two cameras 46, 48) are set in a place wherethe calibration jig 10 can be photographed (step S5). Next, thecalibration jig 10 is photographed by two cameras 46, 48 (step S6). Thephotography by two cameras may be performed either synchronously orasynchronously. Two images photographed in this manner are sent to theprocessing unit 12. Moreover, the camera calibration parameter is firstcalculated for each camera in the processing unit 12 (step S7).Subsequently, the parameter indicating a relative position andorientation between the cameras is calculated (step S8). Thiscorresponds to the calculation of the camera parameter for rectificationwell used in so-called binocular stereo.

It is to be noted that a method of calculating the camera parameternecessary for the rectification after the calculation of the cameraparameters with respect to the respective cameras 46, 48 is described,for example, in the document of Oliver Faugeras or E. Trucco and A.Verri, Introductory Techniques for 3-D Computer Vision, Prentice Hall,1988, pp. 157 to 161 in detail, and is not described here. Additionally,as an important point, in the present invention, the distortionparameter of the lens is also estimated. Therefore, the presentembodiment is different from the documents of Oliver Faugeras and E.Trucco, et al. in that an operation for removing lens distortioncomponents from the camera images is added before performing therectification process, and the rectification is performed.

In general, when the same calibration jig is photographed with aplurality of cameras, ranges in which the left and right cameras canperform the photography differ. Therefore, the calibration marker is notnecessarily photographed simultaneously both in the camera 46 which isthe left camera and the camera 48 which is the right camera. Moreover,there is a problem that a way of photographing the marker or thebackground image differs, and it is therefore difficult to identify themarkers in the left and right images (impart the same number to the samemarker). However, the method of the present invention has a newconstitution and effect that 1) there are clear boundaries (backgroundboundaries 24) in three planes 18 constituting the calibration jig 10,and that 2) large markers 20 b can contribute to an easieridentification of the entire set of calibration markers. Therefore,there is a merit that more robust estimate (calculation) is possible incalculating the calibration parameter of the camera.

Moreover, the image acquisition apparatus 14 constituted of two cameras46, 48 has been described with reference to FIG. 19, but the presentembodiment is not limited to the use of two cameras. As shown in FIG.21, the present embodiment is also applicable to a constitution of theimage acquisition apparatus 14 by the use of a camera 50 including astereo adapter. That is, for the camera 50 including the stereo adapter,as shown in FIG. 22, the images of the calibration jig 10 different fromone another in parallax are photographed in one frame of image 52.Therefore, when one frame of image 52 is cut into two, the image can beobtained as if the image were photographed with two cameras.Additionally, the function and effect are similar to that by the use oftwo calibrations, and are not described here in detail. Therefore, evenwhen the image acquisition apparatus 14 is constituted of the camera 50including the stereo adapter, the calibration apparatus may beconstituted to perform an operation in accordance with an operationflowchart similar to that of FIG. 20.

Fourth Embodiment

In the first to third embodiments, the calibration jig 10 includingthree planes crossing at right angles to one another has mainly beendescribed. However, the present invention is not necessarily limited tothe calibration jig 10 of a corner cube type in which the planes crossat right angles to one another. Basically, any calibration jig may beused as long as the jig is constituted of at least two planes or curvedsurfaces or the three-dimensional positions of the markers in the planeor the curved surface can easily be modeled or registered.

From this viewpoint, the calibration jig may be a calibration jig 54which is a regular tetrahedron as shown in FIG. 23. Alternatively, thejig may also be a calibration jig 56 which has a corner cube shapeincluding three planes having a convex shape, not a concave shape asshown in FIG. 24, and crossing at right angles to one another.Furthermore, as shown in FIGS. 2 to 4, each plane may also have variousshapes. Basically, as described in the first embodiment of the presentinvention, it is important that a marker capable of easily detecting theorigin or the direction vector of the plane be disposed in each plane.When the characteristic marker exists in each plane, and the numberingis facilitated, all the methods described in the first to thirdembodiments are applicable.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details, representative devices, andillustrated examples shown and described herein. Accordingly, variousmodifications may be made without departing from the spirit or scope ofthe general inventive concept as defined by the appended claims andtheir equivalents.

1. A calibration apparatus comprising: a calibration jig comprising: atleast two planes, a plurality of calibration markers having knownthree-dimensional positions, the plurality of calibration markers beingarranged in the at least two planes based on a predetermined rule, andat least one of a plane boundary, a plane boundary group, and a planeboundary curve between the at least two planes in which the calibrationmarkers are arranged, capable of specifying a plane in which acalibration marker is arranged, wherein the at least one of a planeboundary, a plane boundary group, and a plane boundary curve is in acolor different from a color of the background of the at least twoplanes; a calibration marker recognition section configured to recognizein at least one image of the calibration jig obtained by an imageacquisition apparatus the at least one of a plane boundary, a planeboundary group, and a plane boundary curve, to recognize in the at leastone image different planes in the at least two planes as differentregions using the recognized at least one of a plane boundary, a planeboundary group, and a plane boundary curve, and to measure and numberin-image positions of the plurality of calibration markers in the atleast one image; and a parameter estimate section configured to estimateone or more calibration parameters of the image acquisition apparatusbased on the known three-dimensional positions of the plurality ofcalibration markers and the measured in-image positions of thecalibration markers numbered by the calibration marker recognitionsection.
 2. The apparatus according to claim 1, wherein the calibrationjig includes the calibration markers arranged in such a manner that thecalibration markers can be numbered based on the predetermined rule. 3.The apparatus according to claim 1, wherein the calibration jig includesthree planes which cross at right angles to one another.
 4. Theapparatus according to claim 3, wherein at least two types ofcalibration markers are arranged in each plane in which the plurality ofcalibration markers are arranged, and the calibration marker recognitionsection is configured to measure and number in-image positions of theplurality of calibration markers in each plane based on the arrangementrelation of the at least two types of calibration markers.
 5. Theapparatus according to claim 4, wherein the calibration jig includes atleast two types of calibration markers which have different sizes. 6.The apparatus according to claim 4, wherein the calibration jig includesat least two types of calibration markers which have different colors.7. The apparatus according to claim 4, wherein the calibration jigincludes at least two types of markers which have different shapes. 8.The apparatus according to claim 4, wherein in the calibration jig, atleast one type of the calibration markers is disposed in the vicinity ofthe origin at which three planes intersect with one another.
 9. Theapparatus according to claim 1, further comprising a moving sectionconfigured to relatively move one of the image acquisition apparatus andthe calibration jig in order to photograph a plurality of images inwhich the calibration jig is reflected from multiple viewpoints.
 10. Theapparatus according to claim 1, wherein the calibration markerrecognition section performs numbering the calibration markers in eachplane of the calibration jig based on a predetermined order, usesinformation of an image position of a vicinity marker already numberedto predict the position of the marker to be numbered next in the image,and performs one of selecting an optimal candidate from markercandidates detected in accordance with the prediction and judging thatthe optimal candidate does not exist.
 11. The apparatus according toclaim 10, wherein the calibration marker recognition section uses adistance from the prediction position in the selection of the optimumcandidate.
 12. The apparatus according to claim 10, wherein thecalibration marker recognition section uses a distance from theprediction position to judge that the optimal candidate does not exist,and judges that the optimal candidate does not exist, when the distanceexceeds a predetermined threshold value.
 13. The apparatus according toclaim 10, wherein the calibration marker recognition section spirallynumbers the calibration markers.
 14. The apparatus according to claim 1,further comprising a display section configured to superimpose anddisplay a recognition result of the calibration marker recognized by thecalibration marker recognition section upon one of the imagephotographed by the image acquisition apparatus and another image. 15.The apparatus according to claim 1, wherein all the calibration markersof at least one type of a plurality of types of calibration markersarranged on the calibration jig are reflected in the image of thecalibration jig photographed by the image acquisition apparatus.
 16. Theapparatus according to claim 1, wherein all the calibration markers ofat least one type of a plurality of types of calibration markersarranged on the calibration jig are reflected in a middle of the imageof the calibration jig photographed by the image acquisition apparatus.17. The apparatus according to claim 1, wherein the image acquisitionapparatus is image acquisition apparatus including a stereo adapter, andat least one of the images of the calibration jigs photographed by theimage acquisition apparatus including the stereo adapter is photographedin such a manner that all the calibration markers of at least one typeof a plurality of types of calibration markers arranged on thecalibration jig are arranged in a middle of the image.
 18. The apparatusaccording to claim 1, wherein the image of the calibration jigphotographed by the image acquisition apparatus is photographed in sucha manner that a part of one of a boundary and a boundary curve of aplurality of planes constituting the calibration jig is positioned in amiddle of the image.
 19. A calibration apparatus comprising: acalibration jig comprising: at least two planes, a plurality ofcalibration markers having known three-dimensional positions, theplurality of calibration markers being arranged in the at least twoplanes based on a predetermined rule, and at least one of a planeboundary, a plane boundary group, and a plane boundary curve between theat least two planes in which the calibration markers are arranged,capable of specifying a plane in which a calibration marker is arranged,wherein the at least one of a plane boundary, a plane boundary group,and a plane boundary curve is in a color different from a color of thebackground of the at least two planes; calibration marker recognitionmeans for recognizing in at least one image of the calibration jigobtained by an image acquisition apparatus the at least one of a planeboundary, a plane boundary group, and a plane boundary curve, forrecognizing in the at least one image different planes in the at leasttwo a planes as different regions using the recognized at least one of aplane boundary, a plane boundary group, and a plane boundary curve, andfor measuring and numbering in-image positions of the plurality ofcalibration markers in the at least one image; and parameter estimatemeans for estimating one or more calibration parameters of the imageacquisition means based on the known three-dimensional positions of theplurality of calibration markers and the measured in-image positions ofthe plurality of calibration markers numbered by the calibration markerrecognition means.
 20. The calibration apparatus according to claim 1,wherein at least two kinds of calibration markers which are different inat least one of size, shape and color exist in each plane of thecalibration jig, and the calibration marker recognition section isconfigured to label the plurality of calibration markers based on apositional relationship of the calibration markers.
 21. The calibrationapparatus according to claim 19, wherein at least two kinds ofcalibration markers which are different in at least one of size, shapeand color exist in each plane of the calibration jig, and thecalibration marker recognition means labels the plurality of calibrationmarkers based on a positional relationship of the calibration markers.